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(54) Image display 

(57) An interactive virtual endoscopy system 10 in- 
cludes a CT scanner 12 or other non-invasive examina- 
tion apparatus which examines an interior region of a 
subject 14 in an examination region 16 and generates 
data indicative thereof. The data is stored in a volume 
image data memory 20. Using a sequence generating 
computer 22, a human operator 24 generates a se- 
quence of sphere-mappable panoramic views of select- 
ed portions of the CT data along a viewpath in the patient 
14. The sequence generating computer includes a view 
Tenderer 182 for rendering a plurality of views which in 
total cover the entire visual space about a viewpoint on 
the viewpath within the subject. A view compositor com- 
bines the plurality of views into a full image covering the 



entire visual space about the viewpoint. The sequence 
is transferred to a server which processes the data and 
makes it available for remote access. Over a local area 
network (LAN), the data is selectively transferred, based 
on the commands of a remote human viewer, to a re- 
mote viewing computer 34. The data is decompressed 
and mapped into a spherical image for display on a re- 
mote display screen 36. 

Viewers of the sequence of spherical images have 
the liberty to turn at will and view in any direction from 
a particular viewpoint instead of being constrained to a 
look-forward path. In this way. viewers retain the sense 
of order and space which a look-forward series provides 
but with the added capability to investigate completely 
a space from any given viewpoint. 
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Descrlpti n 

[0001 ] The present invention relates to image display. 
It finds particular application in conjunction with provid- 
ing three-dimensional presentations of diagnostic med- 
ical images on remote video monitors and will be de- 
scribed with particular reference thereto. In particular, 
the invention relates to interactive visualization of inter- 
nal cavities in an organism, such as the intestines, bron- 
chi, arteries and the like. However, it is to be appreciated 
that the invention finds application to other areas, such 
as virtual viewing of internal cavities of non-organic sub- 
jects such as pipes and sealed vessels. In addition, it 
will be appreciated that the invention has broad appli- 
cation in conjunction with generating three-dimensional 
diagnostic images from data acquired from various im- 
aging modalities, e.g., by CT scan, magnetic resonance 
imaging and ultrasound. 

[0002] Heretofore, an endoscope is used to view pas- 
sages through or the interior of organs such as the bron- 
chi, oesophagus, stomach, etc. The endoscope is 
threaded into internal cavities within the human body to 
provide real-time, high resolution views of the interior. 
The views may be recorded on video tape for later view- 
ing. Further, the video images may be electronically 
transmitted to remote locations. However, there are a 
number of disadvantages with endoscopic examina- 
tions and video recordings thereof. The endoscope pro- 
vides the operator and remote viewers with only a lim- 
ited field of view without the ability to review in a reverse 
direction. Another problem with endoscopic examina- 
tion is that it is not capable of being used with cavities 
that do not have an opening to the outside. In this regard, 
where possible, certain cavities must be surgically per- 
forated to allow access to the endoscope. Further, be- 
cause endoscopic examination can be uncomfortable or 
even painful, the procedure requires some sedation or 
anesthesia to reduce patient discomfort. 
[0003] The use of computed tomography (CT) scan- 
ners using X-rays also provides limited inspection of in- 
ternal cavities. CT scanners irradiate the planar region 
of a subject from various angles and detect the intensity 
of radiation passing therethrough. From the angle and 
radiation intensity information, two dimensional image 
representations of the plane are reconstructed. Atypical 
image representation includes a 512 by 512 pixel array, 
although smaller and larger arrays are known. In a black 
and white image, each pixel has a corresponding value 
or number which is indicative of the gray scale to be dis- 
played at that pixel. For three-dimensional imaging, a 
plurality of slices are generated, e.g., 60 closely adja- 
cent parallel slices, each of which is represented by a 
512 by 512 array of pixel values. The pixel values of the 
multiple slices are treated as a 512 by 512 by 60 pixel 
array or three dimensions of image data, various planes 
or other surfaces can be defined through the three di- 
mensional data and displayed. However, visualizing the 
interior of a three dimensional cavity from a series of 



slices is difficult. 

[0004] In effort to improve visualization, techniques 
have been developed for generating a three dimension- 
al representation allowing the inspection of an object 

5 along any cutting plane. With these techniques, appro- 
priate planes and surfaces may be selected which per- 
mit viewing of the internal surfaces of cavities in the hu- 
man body. That is, a slice image may be generated 
through a length of the oesophagus. Such a slice can 

10 be processed to reveal the internal surface of one-half 
of the internal cylindrical length of the organ. Generally, 
such three dimensional presentations include a display 
of only the extended surfaces which a viewer would see 
as an internal part of the object through the cut of the 

'5 object by an appropriate plane or surface. 

[0005] To generate the pixel values for display, every 
pixel value of the three dimensional data is examined. 
Each data value is examined to determine whether or 
not it shows in the resultant image. Each data value 

20 which does show is assessed relative to the other data 
values to determine what contribution, if any, it makes 
to the image. None can be readily dismissed as not 
showing. Specifically, air produces a pixel value charac- 
teristic of black. Because air is transparent to the viewer, 

25 values from pixels hidden behind pixels whose values 
are indicative of air show through, hence must be dis- 
played. Analogously, other types of tissue that have 
characteristic pixel values or CT numbers are also de- 
fined as transparent and removed from the view. Hence, 

30 the location of the pixel within the data alone is not de- 
terminative of whether or not the pixel value would show 
in the image. Rather, each pixel value has to be consid- 
ered in the context of its surrounding pixels. This is com- 
putationally very time-consuming. Note that a 512 by 

35 512 by 60 pixel data set contains almost 16 million pix- 
els. Various techniques have been developed, many of 
which are application-specific, i.e., for reducing or iden- 
tifying a subset of all available pixels to project up to the 
cutting surface or viewing screen to determine their con- 

40 tributions. 

[0006] To visualize depth, the angle of the tangent to 
the surface at each point is estimated and shading is 
added in accordance with the angle of the surface tan- 
gent relative to a preselected illumination point. In a 

45 black and white CT image, the shading is added by in- 
creasing the brightness to whiteness of each pixel value 
in proportion to how nearly perpendicular it is to the light 
source, and by increasing the black scale in proportion 
to the degree that the tangential surface faces away 

50 from the light source. For example, a gray scale value 
that is proportional to the sine/cosine of the angle be- 
tween the tangent surface and the light source may be 
combined with each pixel value. 
[0007] Once the three dimensional presentation is 

55 displayed on the screen, it is often advantageous to view 
it from a different orientation. For example, a critical sur- 
face portion may be partially obscured or it may be nec- 
essary to see the back side before starting surgery. For 
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the new viewpoint, the entire process is repeated anew. 
Effectively, all of the data within the three dimensional 
volume is rotated to the appropriate orientation relative 
to the viewing screen, and the contribution of each pixel 
is projected up the plane of the screen for reassess- 
ment. All of the data is rotated or shifted to achieve the 
proper location of the screen relative to the data before 
the data was projected up to the screen. The shifting of 
almost 16 million pixels of data and the interpolation of 
data, where necessary, further adds to processing time. 
[0008] Viewing the CT data with key -frame tech- 
niques provides yet another improvement in the visual- 
ization of internal cavities. In one key-frame technique, 
an operator uses software to move through the data and 
render images from a certain viewpoint and in a certain 
direction. The operator generates a series of such im- 
ages which may then be viewed sequentially as an an- 
imation. Some of the problems with key-frame anima- 
tion are the same as with the video recording of an en- 
doscopy study. The secondary viewer has a limited field 
of view and is restricted to those key-frames selected 
by the initial operator. Further, such studies are overly 
large and cumbersome to use on networks. 
[0009] Another techn ique to visualize internal cavities 
is forward-looking virtual endoscopy which is a type of 
key-frame animation. Using a software package, an op- 
erator selects a path within a cavity or passage. Sequen- 
tial forward-looking views or key-frames are then gen- 
erated along the path and displayed. Forward-looking 
virtual endoscopy also suffers from some of the same 
problems as actual endoscopy. That is, a secondary 
viewer is constrained to a limited field of view and only 
to forward-looking views. In addition, virtual endoscopy 
studies generate large amounts of data which tax net- 
work resources in their transmission, require significant 
time to download and require significant resources for 
their storage. In order to view a study prepared from a 
data set, an entire video sequence must first be down- 
loaded before any image may be viewed. Data com- 
pression of a virtual endoscopy video can speed trans- 
mission and smooth playback over a network but it also 
degrades noticeably the quality of the images. 
[0010] Forward-looking virtual endoscopy may be en- 
hanced by capturing images outside of key-frame ani- 
mation. In this technique, the radiologist leaves the key- 
frame animation paradigm of image capture to manually 
reorient for capture of structures which are, for example, 
perpendicular to a viewpath through an organ. This also 
has its problems. 

[0011] First, it is difficult and time-consuming to cap- 
ture views while constructing a sequence of virtual en- 
doscopy images. Second, some visio-spatial context of 
the organ is lost in the frequent reorientations necessary 
for adequate coverage of the organ structure and sub- 
sequent reorientations of the camera to recover the orig- 
inal path. Thus, there is a possibility that the radiologist 
charged with capturing sequences will not capture some 
views crucial to diagnostic confidence due to the diffi- 



culties inherent in manual navigation and capture of im- 
ages in virtual endoscopy datasets and subsequent re- 
orientations of the rendering camera. In particular, the 
back surfaces of structures blocking passages require 
5 special attention. Failure to capture fully and accurately 
the context of the structure is possible if capture is left 
entirely to manual camera orientation. 
[001 2] In accordance with the present invention, a di- 
agnostic imaging apparatus presents a three dimen- 
10 sional image representation. The apparatus includes an 
image data memory for storing image data indicative of 
a three dimensional volume and an image data memory 
accessor for selectively accessing the stored image da- 
ta. A view renderer renders a plurality of views which in 
is total cover the entire visual space about a viewpoint 
within the three dimensional volume. A view compositor 
combines the plurality of views into a full image covering 
the entire visual space about the viewpoint. A mapper 
maps the full image into a spherical panoramic image. 
20 a video processor displays a portion of the spherical, 
panoramic image on a display. 

[0013] In accordance with another aspect of the 
present invention, an internal spherical view from a 
viewpoint within a three dimensional array of image data 
25 is generated using a computer. A plurality of two-dimen- 
sional arrays are generated which in total cover the en- 
tire spherical space about the viewpoint. At least one of 
the plurality of two-dimensional arrays is divided into a 
plurality of first polygon arrays. The plurality of first pol- 
30 ygon arrays are scaled into a plurality of second polygon 
arrays. The plurality of second polygon arrays and a por- 
tion of the plurality of two-dimensional arrays are com- 
bined to form a full two-dimensional array covering the 
entire spherical space about the viewpoint. The full two- 
35 dimensional array is mapped into a spherical view. At 
least a portion of the mapped, full, two-dimensional ar- 
ray is displayed as image pixels in a human-readable 
display. 

[0014] Ways of carrying out the invention will now be 
40 described in detail, by way of example, with reference 
to the accompanying drawings, in which: 

Figure 1 is a diagrammatic illustration of an inter- 
active viewing system for generating virtual endos- 
45 copy studies of medical diagnostic data in accord- 
ance with the present invention; 

Figure 2 A and Figure 2B are partial detailed dia- 
grammatic illustrations comprising the whole of the 
50 sequence generating computer 22 of Figure 1 ; 

Figure 2C is a full diagrammatic illustration of the 
server 22 and remote computer 34 of Figure 1 : 

55 Figure 3 is a diagrammatic illustration of the bron- 
chial tubes of a patient which are to be viewed by 
interactive viewing system; 
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Figure 4 is a enlarged view of the bronchial tubes 
of Figure 3 showing in more detail a viewpath 
through the bronchial tubes; 

Figure 5A is a diagrammatic illustration of an ar- 
rangement of six flat views to generate a high-dis- 
tortion spherical image about viewpoint; 

Figure 5B is a diagrammatic illustration of an ar- 
rangement of 26 flat views to generate a low-distor- 
tion spherical image about viewpoint; 

Figure 6A illustrates the dividing and cropping of the 
26 views of Figure 5B to obtain a full, flat image for 
a low-distortion spherical panoramic view; 

Figure 6B illustrates the arrangement of the 26 
views of Figure 5B to obtain a full, flat image for a 
low-distortion spherical panoramic view; and, 

Figure 7 is a partial detailed diagrammatic illustra- 
tion of a second embodiment of a portion of the se- 
quence generating computer 22 of Figure 1; 

Figure 8 illustrates the dividing and arrangement of 
the six views of Figure 5A to obtain a full, flat image 
for a high-distortion spherical panoramic view; 

Figure 9 is a diagrammatic illustration of a repre- 
sentative display of the spherical image, the axial, 
sagittal and coronal views associated with the im- 
age, patient and view information, and patient diag- 
noses fields that is displayed on the display monitor 
36 of Figure 1. 

[0015] With reference to Figure 1, an interactive vir- 
tual endoscopy system 10 includes a CT scanner 12 or 
other non-invasive examination apparatus which exam- 
ines an interior region of a subject 14 in an examination 
region 16 and generates data indicative thereof. The CT 
data is stored in a volume image data memory 20. Using 
a sequence generating computer 22, a human operator 
24 generates a sequence of sphere-mappable pano- 
ramic views of selected portions of the CT data along a 
viewpath in the patient 14. The sequence is transferred 
to a server 26 which processes the data and makes it 
available for remote access. Over a local area network 
(LAN) 30, the data is selectively transferred, based on 
the commands of a remote human viewer 32, to a re- 
mote viewing computer 34 where the data is decom- 
pressed and displayed on a remote display screen 36. 
[0016] With continuing reference to Figure 1, more 
specifically, the CT scanner includes a subject support 
or patient couch 40 for supporting a patient or other sub- 
ject received in the examination region 16. An irradiating 
means 42, such as an x-ray tube, irradiates the patient 
with x-rays or other penetrating radiation. A radiation re- 
ceiving means, such as radiation detectors 44, receive 



diagnostically encoded radiation which has traversed 
the examination region. In the preferred embodiment, 
the x-ray tube generates a fan-shaped beam of x-rays. 
The fan-shaped beam of x-rays passes through the sub- 
5 ject and impinges upon an arc of the radiation detectors 
44. The x-ray tube is mounted for rotation by a motor or 
other rotating means around the examination region, 
such that the subject is irradiated from a multiplicity of 
directions. The radiation detectors are positioned either 
10 in a stationary ring surrounding the examination region 
or in an arc which rotates with the x-ray tube to receive 
the radiation which has traversed the subject. The pa- 
tient is moved along a longitudinal axis of the CT scan- 
ner either continuously for spiral scanning or incremen- 
ts tally, to generate a multiplicity of slices. Optionally, a wid- 
er x-ray beam may be generated and a plurality of rows 
of detectors may be provided such that a plurality of slic- 
es are generated concurrently. 

[0017] An image reconstruction processor 50 sam- 

20 pies the radiation from the radiation detectors 46 and 
reconstructs an image representation therefrom. It will 
be noted, that each time a radiation detector 16 is sam- 
pled, the intensity of sampled radiation represents a 
sum of the radiation absorptive properties of tissue lying 

25 along a ray or path between the x-ray source (at the time 
of the sampling) and the sampled detector. Because the 
intensity of radiation at the x-ray source is known and 
the intensity of detected radiation is known, the line in- 
tegral of the material in the region 16 can be estimated 

30 such that it represents the line integral of the radiation 
absorption or attenuation along each ray. The recon- 
struction processor 50 extrapolates or interpolates as 
necessary to generate CT numbers corresponding to 
each of a three dimensional array of voxels. 

35 [0018] An original volume image data memory 20 
stores the reconstructed volumetric data. Each of the 
CT numbers for the data can be conceptualized as heing 
addressable by its coordinate location along three or- 
thogonal axes, e. g. x, y, and z axes of the examined vol- 

40 ume. For example, the pixel values may be triply sub- 
scripted variables, such that the corresponding pixel val- 
ue is accessible based on its x, y, z spatial position in 
the volume. In this manner, pixel values representing a 
regular distribution over an imaged volumetric region 

45 are generated and stored. 

Path Selection 

[0019] With continuing reference to Figure 1 and ref- 
50 erence to Figure 2A, the sequence generating computer 
22 includes an operator control 60 to be used by the 
human operator 24, typically, a radiologist, in order to 
select a desired path through an organ. The operator 
control 60 has the appropriate keys, joy stick, trackball, 
55 or mouse arrangement to permit the operator 24, to se- 
lectively access certain data in the volume image data 
memory 20 and cause it to be displayed on the screen 
of a sequence computer video display 62. This selection 
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process is performed whife viewing the CT data on the 
video display 62 in any of a number of formats such as 
slice images through the body, three-dimensional pro- 
jections of external views of organs of interest, key- 
frame animation techniques and the like. 
[0020] With continuing reference to Figure 2 A and ref- 
erence to Figure 3, in the preferred embodiment, the op- 
erator uses key-frame animation techniques to aid in the 
selection of a desired viewpath 70 in the bronchial tubes 
72 of the patient 14. The illustrated example shows an 
external view of the patient and bronchial tubes. How- 
ever, it is appreciated by those skilled in the art, that rath- 
er than an external view, internal views arc generated 
by the key frame animation technique. Using the oper- 
ator control 60, the operator or radiologist moves step- 
wise through the volumetric data to find the bronchial 
tubes. The operator inputs position and direction data 
into the operator control 60. The operator control 60 al- 
lows selection and adjustment of the field of view (FOV) 
of the key-frames as well other imaging parameters 
such as shading, colouring, and the like. A frame ren- 
dering processor 74 accesses certain data in the volume 
image data memory 20 by means of an image memory 
access processor 76. The frame rendering processor 
controls a ray projection processor 78 which calculates 
a plurality of rays from a viewing plane at the selected 
position and in the selected direction. The viewing plane 
is represented by the screen of the video terminal 62. 
The rays are projected from the viewing plane through 
the three-dimensional array of the volume data. The 
three-dimensional array data is characterized by grey 
scale values corresponding to each of a three-dimen- 
sional array of data points or voxels. The organs or tis- 
sues to be viewed are typically buried within the data. A 
ray from each pixel of the viewing or image plane is pro- 
jected orthogonal to the viewing plane into the data. An 
air subtractor 82 subtracts or ignores pixels having the 
value of air along the calculated rays. The calculated 
rays continue until they either intersect a pixel indicative 
of a tissue or reach an endpoint of the three-dimensional 
array of data. From the depth at which each ray inter- 
sects a tissue and whether a ray intersects a tissue at 
all, a mapping processor 84 generates a map of the sur- 
face contour of the tissues when viewed from the view- 
ing plane. 

[0021] A colouring processor 86 adds colour to the 
map depending on the grey scale level or CT value. This 
permits easier identification of certain tissues. 
[0022] To give meaning to the surface contour, a 
shading processor or routine 90 adds shading to the ex- 
posed surfaces of the key-frame to give the effective 
depth in the direction perpendicular to the screen. The 
shading processor adds shading to the map of the sur- 
face contour as if a light source existed at the viewing 
plane. The shading processor uses any one of many 
methods for adding shading including depth shading, 
depth gradient shading, gray scale level gradient shad- 
ing, and rendering with polygonal shapes. 



[0023] Once each key frame images has been gener- 
ated and suitably enhanced, it is stored in a key-frame 
memory 92. An optional smoother 94 generates a 
smooth path between sequential key frames. In the pre- 
s ferred embodiment, the smoother uses the technique of 
cubic splines. The sequence of smoothed key-frames 
are stored in an animation memory 96. A video proces- 
sor 100 formats the data for display on the sequence 
computer video display 62. 

[0024] With continuing reference to Figures 2A and 3, 
and further reference to Figure 4, while viewing the key- 
frame images of the CT data on the video display 62, 
the operator selects the desired viewpath 70, via the op- 
erator control 60 and a graphical interface 102, by des- 
ignating at least a beginning path point 110 and an end 
path point 112. In the example shown in Figure 3, the 
operator selects a cavity or channel 114 within the bron- 
chial tubes 72 between the epiglottis or beginning path 
point 110 and a point deep in the lung or the end path 
point 112. If desired, a second set of beginning and end 
path points 116,118 may be selected to include a branch 
120 within the organ. Additionally, the operator may 
mark or tag, via the operator control, certain tissue, such 
as a tumor 142, for easier identification by the remote 
viewer. 

[0025] With continuing reference to Figure 4 and ref- 
erence to Figure 2B, upon path selection, a cavity loca- 
tor 130 scans the data between and around the path 
points 110, 112 to identify all CT pixel data having the 
image intensity, or Houndsfield value, of air which lies 
within the desired channel 114. In other embodiments, 
the cavity locator is programmed to scan for other uni- 
form media that are in the desired cavities such as blood 
or cerebrospinal fluid. Based on the locus of pixels hav- 
ing the image intensity of air, a centroid path processor 
132 calculates a centroid path generally midway in the 
channel. This centroid path has already been designat- 
ed as the viewpath 70. 

[0026] With continuing reference to Figure 2B and 
Figure 4, a path-dividing processor 134 divides the cen- 
troid path 70 into equal length segments 1 36. The length 
of the segments may be varied by the operator depend- 
ing on the desired resolution of the final sequence. End- 
points of the segments 136 define viewpoints 140. 
These viewpoints are the central or vantage points of 
the spherical panoramic views that are to be generated. 
[0027] In another embodiment, rather than allowing 
the path-dividing processor to define a multitude of view- 
points, the operator may selectively define one, two or 
more viewpoints for a sequence. For example, with ref- 
erence to Figure 4, by selecting only two viewpoints 1 42, 
the operator and remote viewer may focus the sequence 
or study on one area of concern, such as a tumor 146 
in the bronchial tubes. 

[0028] In the preferred embodiment of the present in- 
vention, the viewpoints 140 are low-distortion view- 
points. These low-distortion viewpoints 1 40 are to be the 
centres of spherical panoramic views generated from 26 
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flat images. In another embodiment of the invention, the 
viewpoints 140 are high-distortion viewpoints which are 
to be the centres of spherical panoramic views gener- 
ated from only 6 flat images. Because more images are 
incorporated into the spherical panoramic views around 
low-distortion viewpoints, these viewpoints provide 
high-precision, high-detail examinations. In contradis- 
tinction, spherical images around high -distortion view- 
points do not provide as much precision or detail. Of 
course, the generation of 26-tmage spherical views re- 
quires significantly more computing power than the gen- 
eration of 6-image spherical views. 

Rendering 

[0029] With continuing reference to Figure 2B and 
Figure 4, to ensure smooth transitions between each 
rendered spherical image, a tangent vector processor 
and associated memory 150 calculates and stores a 
tangential vector 152 to the viewpath 70 for each view- 
point. The tangential vector is defined as z'. For each 
successive viewpoint, an orientation processor and as- 
sociated memory 154 defines and stores a vector x' 
which is orthogonal to z'. In Figure 4, x' is directed or- 
thogonal to the sheet. The orientation processor mini- 
mizes the angular deviation of the x' vectors between 
two neighbouring viewpoints. With continuing reference 
to Figure 4 and reference to Figures 5A and 5B, the z' 
and x' vectors define the orientation of the sets of 6 or 
26 flat views 160, 162 that arc to be generated. As an 
aid to this visualization, Figure 4 shows cubes 158 
where the front wall is defined by z' and the right side 
wall (with inexact perspective) is defined by x'. This 
process ensures that, during viewing, no two neighbour- 
ing spherical images undergo a severe twist in orienta- 
tion which could be perceived by the viewer as a shift in 
distortion when moving between the images. 
[0030] In another embodiment, the tangential proces- 
sor and orientation processor are absent from the sys- 
tem, in this embodiment, the x' and z' orientations of the 
6 or 26 flat views simply mirror the x and z orientation 
of the stored volume data. 

[0031] With continuing reference to Figures 5 A and 
5B, by way of background, an inherent limitation in cre- 
ating spherical panoramic views is that the source im- 
ages are flat, that is, the images are projections of a 
scene onto a plane. The panoramas created by flat im- 
ages are therefore at best a multi-faceted sphere. A bet- 
ter approximation of the spherical panorama is achieved 
with smaller facets, or equivalently, source images with 
a smaller field of view (FOV). Using smaller facets, how- 
ever, also increases the number of source images re- 
quired to span the spherical view around a point. Fur- 
ther, the greater the number of source images, the great- 
er is the amount of computer processing that is needed 
to create the spherical views. 

[0032] As indicated above, the advantage of the 
26- image spherical panorama is a more accurate ap- 



proximation of the spherical world. As seen in Figures 
5A and 5B, the 26-sided polygon 162 formed by the set 
of 26 images more closely approximates a sphere than 
the cube 1 60 formed by the set of 6 images. Accordingly, 
s with continuing reference to Figures 2B and 5B, the se- 
quence generating computer 22 creates a sequence of 
spherical panoramas using 26 source images per view- 
point. 

[0033] With continuing reference to Figure 5B, and 
10 reference to Figure 6A and Figure 8, source images 1 70 
are acquired as square images with identical field of 
view along both horizontal and vertical directions. In the 
preferred embodiment, the pixel dimensions of the 
source images for the 26-face approach 162 are 256 x 
is 256 and the FOV is 45. 

[0034] With reference to Figure 2B and Figure 4, for 
every viewpoint 140, a 26-frame rendering processor 
182 controls the rendering of 26 views around a view- 
point. With reference to Figure 5B, each set of 26 views 
162 covers the entire mappable space about a view- 
point. With reference to Figure 2B, the rendering proc- 
essor 182 controls a ray projection processor 184 which 
calculates a plurality of rays from a viewing plane at the 
selected viewpoint and in a direction dependent on the 
x' and y' vectors for the selected viewpoint. The rays are 
projected from the viewing plane through the three-di- 
mensional array of the volume data. A ray from each 
pixel of the viewing or image plane is projected orthog- 
onal to the viewing plane into the data. 
[0035] During rendering ot the set of 26 images, an 
air subtractor or routine 186 identifies those addressed 
pixels in the three-dimensional image volume which rep- 
resent air or other selected tissue types which arc not 
to be displayed. If the pixel value is representative of air, 
the ray projector 184 moves one pixel step in the pro- 
tection direction, i.e., the direction perpendicular to the 
viewing screen, further into the image data and retrieves 
the next corresponding pixel. This pixel is again com- 
pared with the air or other preselected tissue values to 
determine whether it is to be displayed or not. This proc- 
ess is iteratively repeated until a memory cell or pixel 
value of the appropriate range to be displayed is located. 
From the depth at which each ray intersects a tissue and 
whether a ray intersects a tissue at all, a mapping proc- 
essor 1 90 generates a map of the surface contour of the 
tissues when viewed from the viewing plane. It is to be 
appreciated by those skilled in the art that other tissue 
rendering techniques may be used in the present inven- 
tion such as volumetric compositing and the like. 
[0036] A colouring processor 192 adds colour to the 
map depending on the grey scale level or CT value. This 
permits easier identification of certain tissues. 
[0037] For a more lifelike or realistic three dimension- 
al presentation, a shading processor or routine 194 pro- 
vides shading values to each pixel. More specifically, an 
angle or position of illumination origination is defined. 
Rays are projected from the point of the illumination 
source to the defined volume, with the air or other se- 
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lected tissue removed. The retrieved pixel value is made 
darker or lighter in accordance with th distance and 
surface tangent relative to the light source. Surfaces 
which are obstructed from the light source by other sur- 
faces are shaded most darkly. The degree of shading 
which is added may be selectively adjustable from the 
operator control which can be conceptualized as bright- 
ening or darkening the illumination source. Similarly, the 
illumination source can be repositioned and the shading 
process repeated in order to illustrate other parts of the 
displayed object better. In the preferred embodiment, 
the illumination source is at the viewing plane of each 
rendered flat image. Each rendered image stored in an 
image memory 200. 

C mpositlnq 

[0038] With continuing reference to Figure 2B, and 
reference to Figures 6A and 6B, after a set of 26 square, 
flat, partial images 238 is created, a compositing proc- 
essor 240 crops, scales and combines the set of partial 
images to create a full, flat, output image 242 which rep- 
resents a spherical panoramic view. The special com- 
positing technique employed by the compositing proc- 
essor increases resolution and decreases distortion in 
the resultant spherical panoramic image. 
[0039] The composited, full, flat, output image 242 is 
similar in nature to a rectangular world map of the earth. 
Of the set of 26 partial images 238, eight equatorial fac- 
es or partial images 244 form the equator of the spher- 
ical image. With reference to Figure 6A, above and be- 
low the equatorial faces are eight northern hemispheri- 
cal faces 250 and eight southern hemispherical faces 
252. A top or northern pole face 254 and a bottom or 
southern pole face 256 completes the 26-sided spheri- 
cal approximation. 

[0040] With reference to Figures 2B, 6 A and 6B, an 
equatorial view processor or routine 260 places the 8 
equatorial views 244 side by side forming the middle 
portion of the full output image 242. To appropriately size 
the partial images to form the 26-sided sphere, a crop- 
ping processor 262 crops (at the dotted lines of Figure 
7 A) the northern and southern hemispherical faces 250, 
252 into trapezoidal shapes 264 (with sides represented 
by dotted lines). Further, the cropping processor 262 
crops the top and bottom faces 254, 256 into an octag- 
onal shape 270 (of Figure 5B). A octagonal view divider 
272 splits the octagonal faces 270 into eight triangles 
274 such that each side of the octagons 270 is the base 
of a triangle 274. 

[0041] A scaling processor 280 scales the northern 
and southern hemispherical trapezoidal ly cropped 
views 264 into 256 x 256 pixel rectangles 282. Further, 
the scaling processor 280 scales each triangle 270 of 
the octagonal views 270 into 256 x 1 28 pixel rectangles 
286. To achieve these transformations, the scaling proc- 
essor inserts additional interpolated data between se- 
lected data points to fill out the trapezoidal and triangular 



arrays and transform them into rectangular arrays. In the 
preferred embodiment, the scaling processor uses the 
technique of cubic interpolation. However, it is to be ap- 
preciated that other interpolation techniques may be 

£ used. An arranger 290 appropriately places the scaled 
triangles above and below the cropped and scaled hem- 
ispherical faces 282. Optionally, the arranging proces- 
sor averages data in the overlapped edges. The result- 
ant full, flat images from the compositor 240 are stored 

10 in a sequence memory 300. In the preferred embodi- 
ment, the composited, full image is an array of 2048 x 
1024 pixel data points. 

[0042] In the preferred embodiment, the full, flat im- 
ages comply with the DICOM (Digital Imaging and Com- 

15 munications in Medicine) standard used by most hospi- 
tals. However, it is to be appreciated that the image data 
may be conformed to other imaging standards. A field 
processor 302 assigns spatial information to each of the 
generated full images to assist in the later ordering and 

20 display of the images. Optionally, if the operator marks 
or tags a specific tissue, such as a tumor, the field proc- 
essor may associate the three-dimensional coordinate 
of the tissue with the sequence of images. The entire 
sequence of images and associated fields containing 

25 spatial and other information is stored in the sequence 
memory 300. 

[0043] The sequence of full views provides advantag- 
es not seen in the prior art. As contrasted with key-frame 
based animation, there is no need to constantly reorient 
30 oneself for capture of views not forward-facing along the 
path. Further, the back faces or surfaces of tumors or 
other structures are naturally rendered as part of the 
spherical views permitting easy capture and inspection 
of such views. 

35 [0044] In another embodiment of the invention, the 
sequence generating computer creates full, flat images 
from 6 views about each viewpoint. These viewpoints 
are considered high-distortion viewpoints. With refer- 
ence to Figures 5A, 7 and 8, the sequence generating 
computer 22 acquires six source images 170 for each 
viewpoint. The advantage of this strategy is lesser com- 
puter processing time. The source images 170 are ac- 
quired as square images with identical field of view 
along both horizontal and vertical directions. The FOV 

4S and the pixel dimensions of the source images for the 
6-face approach 1 60 are 90 and 51 2 x 51 2, respectively. 
[0045] With continuing reference to Figure 7 (in which 
elements identical to those in Figure 2B are shown with 
the same reference numerals, and in which elements 

so analogous to those in Figure 2B are shown with the 
same but primed (') reference numerals) and reference 
to Figure 4, for every viewpoint 140, a 6-frame rendering 
processor 182* controls the rendering of 6 views about 
each viewpoint. With reference to Figure 5A, each set 

55 of 6 views 1 60 covers the entire mappable space about 
a viewpoint. With continuing reference to Figure 7, the 
rendering processor 182* control a ray projection proc- 
essor 184. During rendering of the 6-image set, an air 
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subtractor or routine 1 86 identifies those addressed pix- 
els in the three-dimensional image volume which repre- 
sent air or other selected tissue types which are not to 
be displayed. From the depth at which each ray inter- 
sects a desired tissue and whether a ray intersects a 
tissue at all, a mapping processor 190 generates a map 
of the surface contour of the tissues when viewed from 
the viewing plane. A colouring processor 192 adds col- 
our to the map depending on the grey scale level or CT 
value. A shading processor or routine 194 provides 
shading values to each pixel. Each image and set of 6 
images are stored in an image memory 200'. 
[0046] With continuing reference to Figure 7 and ref- 
erence to Figure 8, after a set of 6 flat square views 170 
are created, a compositing processor 240' combines the 
set of square images to create a full, flat, output image 
206 which represents a spherical panoramic view. The 
composited image is similar in nature to a rectangular 
world map of the earth. 

[0047] With reference to Figures 5 A and 8, the six flat, 
square views are analogous to a cubic room which has 
four wall views 210, a ceiling view 212 and a floor view 
214. With continuing reference to Figures 5A and 8 and 
reference to Figure 7, a wall processor or routine 260' 
places the four wall views 210 side by side to form the 
middle portion of the full output image 206. A view di- 
viding processor 272' divides the ceiling view 212 and 
floor view 214 into four triangles 224 each. Of course, 
each triangle of the ceiling and floor views is associated 
with the wall to which its base, when viewed as the cubic 
set of six images, is adjacent. A scaling processor 280" 
scales each triangle into a rectangular array of 512 x 
256 pixels 230. To achieve this transformation, the scal- 
ing processor inserts additional interpolated data be- 
tween selected data points to fill out the triangular array 
and transform it into a rectangular array. An arranging 
processor 290' appropriately aligns and/or overlaps the 
edges of the wall views 210 and the ceiling and floor 
views 212, 214 to form the final output image 206. The 
arranging processor averages the data in the over- 
lapped edges. The composited, full image is an array of 
2048 x 1024 pixel data points. 

[0048] A field processor 302 assigns spatial informa- 
tion to each of the generated full images to assist in the 
later ordering and display of the images. The entire se- 
quence of images and associated fields containing spa- 
tial information is stored in the sequence memory 300. 

Additional Views 

[0049] Returning to the embodiment described in Fig- 
ure 2B, to provide the remote viewer with visio-spatial 
context an additional view rendering processor 310 
renders axial, sagittal and coronal views of any of the 
entire body, portion of the body, external three-dimen- 
sional view of the organ of interest or any other desired 
view. These views are to be displayed alongside a 
spherical image on the remote video terminal. 



[0050] The field assignor 302 assigns spatial informa- 
tion to the views to permit them to be coordinated with 
the spherical panoramic images. The views are then 
stored in the sequence memory 300. 

5 

Transfer and Display 

[0051] With reference to Figure 1 and Figure 2C, to 
make the sequence available over the network 30 the 
10 data from the sequence memory is transferred to the 
designated server 26. In the preferred embodiment, the 
server supports a local area network in a hospital and 
is accordingly DICOM compliant. However, it is to be ap- 
preciated that the server could also support a wide area 
is network including the World Wide Web. Further, it is to 
be appreciated that data protocols other than DICOM 
may be used with the present invention. 
[0052] With continuing reference to Figure 2C, upon 
reception of the sequence data from the sequence 
memory 300, a server control 320 directs a parsing proc- 
essor 322 to parse the spatial information fields associ- 
ated with the sequence of full images as well as the ax- 
ial, coronal and sagittal images. Based on the results of 
the parsing, a script node processor 326 creates a single 
script file containing all spatial and linking information 
for the sequence of full images as well as the associated 
axial, sagittal and coronal views. In the preferred em- 
bodiment, this script file is compliant with VMRL2.0 (Vir- 
tual Reality Modelling Language) for viewing with a 
spherical viewer application. 

[0053] Although it is possible to link the display of one 
spherical image to another via a simple URL (Uniform 
Resource Locator) link, this is undesirable because the 
download and parsing of each text file takes a significant 
amount of time. Consequently, it takes a significant 
amount of time to display an image. Once an image has 
been viewed, it and its script file are cached on a hard 
disk of the remote computer system. However, to review 
the image, the script file must be re-parsed. Parsing time 
depends on processor speed, random access memory 
and the text file size and may take from ten to sixty sec- 
onds. 

[0054] The server control 320 also directs a compres- 
sion processor 340 to parametrically compress each of 
the spherical images. In the preferred embodiment, the 
compression processor compresses the data into JPEG 
files with an 8 to 1 ratio. However, in other embodiments, 
compressed and non -compressed files, such as BMP 
and GIF, and other compression ratios are used which 
may depend on the capabilities of the system and the 
acceptable level of distortion caused by compression- 
decompression. The compressed images and the script 
file are stored in a server output memory 344. 
[0055] Parsing and compression of the virtual endos- 
copy sequence permits more rapid transmission of the 
data over networks and more efficient use of network 
resources. 

[0056] However, as bandwidth widens and network 
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resources improve, it is contemplated that compression 
of the data and indeed the server may be omitted in 
some viewing systems. 

[0057] The data in the server output memory 344 is 
made available to a plurality of remote computers 34 via 
the local area network 30. The remote viewer 32 inputs 
commands to the remote computer 34 to call up the de- 
sired virtual endoscopy sequence. The commands are 
entered to a remote operator control 350 which includes 
a keyboard and trackball. It is to be appreciated that oth- 
er input devices may be used such as a joystick, a 
mouse, a touchpad and the like. A web browser Java 
applet, processor or application 354 on the remote com- 
puter accesses the script file in the server output mem- 
ory 344 and downloads it to a hard drive or cache mem- 
ory 438 on the remote computer. Typically, the address 
of the script file is embedded in another text page with 
an html (hypertext modelling language) file extension. 
When the page is opened in the web browser Java ap- 
plet 354, the applet launches a virtual reality viewing 
processor or application 362. The viewer application 
362 then begins the downloading of the compressed full 
images of the sequence. The viewer application in- 
cludes a decompressing processor or routine 358 to ex- 
pand the compressed full image files. 
[0058] A video processor 366 displays the images on 
the video screen 370 (see also Figure 9). The viewer 
application 362 permits the viewing of the entire visual 
space from each viewpoint of the sequence. It is to be 
appreciated that other browsers or applications may be 
used in conjunction with more limited views of visual 
space, such as cylindrical views. The viewer application 
362 includes a node accessing processor or routine 374 
to access a selected node or viewpoint and the full, flat 
image associated therewith. A spheremapping proces- 
sor 378 maps the flat image into a 360 x 180 degree 
spherical panorama. 

[0059] The viewing application or processor 362 sup- 
ports two rotational degrees of freedom: pitch and yaw. 
Pitch and yaw are controlled by an environment map- 
ping processor 382 which alters the mapping of the dis- 
played environment. In this process, the panorama is 
re-projected onto the spheric surface surrounding the 
remote viewer thereby displaying unseen portions of the 
spherical image. Taking the plane of the screen 370 of 
the monitor 36 as a reference origin, the remote operator 
control 350 and viewing application 362 enable the re- 
mote viewer to rotate pitch and yaw to selectively view 
any portion of the spherical image about its viewpoint. 
[0060] The operator control 350 and viewing applica- 
tion 362 permit the remote viewer to move along the 
viewpath in a forward or backward direction while se- 
lecting the viewing direction. This provides unique view- 
ing techniques such as panning down the wall of the 
bronchial tube while looking directly at the wall. This is 
not possible with forward-looking techniques where the 
viewer obtains only a peripheral view of the wall. 
[0061] To provide movement along the viewpath, a 



path motion processor 394 scales or translates the dis- 
played view. For example, when the viewer is looking 
forward while moving forward along the viewpath, the 
path motion processor enlarges the scale of the forward 
5 view, i.e., zooms in, to give the illusion that the viewer 
is moving forward. Analogously, when the viewer is look- 
ing forward while moving backward, the path motion 
processor reduces the scale of the forward view to give 
the illusion that the viewer is moving backward. To give 
10 the illusion of moving forward along the viewpath while 
looking sideways, the path motion processor translates 
image data from the side of the displayed portion of the 
full image onto the screen and alters the mapping of the 
displayed view. When movement along the path reach- 
is es the limits of the displayed spherical view, the path 
motion processor triggers the display of the next spher- 
ical view. 

[0062] In addition, the remote operator control 350 
and viewing application 362 allow the remote viewer, by 
means of an image adjusting processor or routine 398 
to adjust the image parameters such as field of view (i. 
e., zoom in and out), and the like. As the remote viewer 
redirects or otherwise changes the view, the video proc- 
essor 366 continues to cause the video display 36 to 
display a three dimensional presentation of the interior 
of the cavity onto the screen. 

[0063] With continuing reference to Figure 2C and ref- 
erence to Figure 9, the viewing processor 362 directs 
the display of the sphere-mapped image 400 onto the 
screen 370. An additional view processor or routine 402 
in the viewer application 362 provides display of an ax- 
ial, a sagittal, and a coronal view 406, 408, 410 of the 
silhouette of the patient and of an external view of the 
bronchial tubes or organ of interest. In other embodi- 
ments, the additional view processor supports the dis- 
play of orthogonal views of the patient or a region of the 
patient such as a silhouette of the patient, an external 
view of the organ of interest, slice images through the 
body, and the like. The additional view processor incor- 
porates an indicator 414 onto each of the additional 
views 406, 408, 410 to dynamically show the location 
and orientation of the viewpoint of the correspondingly 
displayed spherical image 400. In the preferred embod- 
iment the indicator has green cross-hairs which indicate 
the location of the viewpoint. A red dot shows the view- 
ing direction by revolving about the centre of the cross- 
hairs in correspondence with the view direction. The red 
dot stops within one of the quadrants formed by the 
cross-hairs to show the angular viewing direction within 
the plane of the additional view. 

[0064] A patient field processor or routine 418 in the 
viewer application provides the display of patient text 
and view information fields 422 for accurate presenta- 
tion of study data. A navigation aid processor 426 dis- 
plays an orientation pointer 430 in the display of the 
spherical image 400. The orientation pointer is a visual 
aid for interpretation of the context of the image data. 
Optionally, the viewer application permits a remote 
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viewer, such as radiologist or disease specialist, to view 
and enter diagnostic opinions or other annotations into 
a patient diagnoses data file 434 for association with the 
images and the sequence. Other remote viewers ac- 
cessing the server would then be able to view the anno- 
tations associated with the images. To assist the remote 
viewers in quickly locating particular annotations, the 
viewer application supports an option to find or jump to 
a selected word, number or alphanumeric combination, 
or position and viewer orientation in the sequence of im- 
ages or study. 

[0065] Viewed images are stored in the cache mem- 
ory 438 for more easy retrieval should the viewer choose 
to move backwards in the sequence. A graphical inter- 
face 444 permits simultaneous display and exchange of 
data between the output of the viewing processor and 
the input of the remote viewer. 

[0066] Viewers of the sequence of spherical images 
have the liberty to turn at will and view in any direction 
from a particular viewpoint instead of being constrained 
to a predetermined look-forward path. In this way, view- 
ers retain the sense of order and space which a look- 
forward series provides but with the added capability to 
investigate completely a space from any given view- 
point. Further, the viewers employ an intuitive interface 
(point and click) instead of a frustrating and time con- 
suming play-reverse-replay interface as with to video. 
[0067] The sequence of spherical images of the 
present invention is selectively downloaded as needed, 
image by image. This enables more productive use of 
network time by radiologists in contradistinction to tra- 
ditional video for which an entire sequence is download- 
ed before being viewed. 

[0068] The above described embodiments thus pro- 
vide an inter-active viewing system for generating virtual 
endoscopy studies of medical diagnostic data with a 
continuous sequence of spherical panoramic views and 
viewing the studies over networks. One advantage is 
that it provides full, unobstructed spherical panoramic 
views from within a three-dimensional array of image 
data. Another advantage is that it facilitates remote 
viewing and diagnoses of internal cavities as represent- 
ed in the three-dimensional array of image data. A fur- 
ther advantage is that it permits greater diagnostic con- 
fidence for secondary viewers of the three-dimensional 
array of image data. Still another advantage is that it 
permits rapid transmission of the images to remote lo- 
cations over networks and more efficient use of network 
resources. Further, although an endoscope examina- 
tion is mostly non-invasive, the procedure still requires 
some sedation or anesthesia to reduce patient discom- 
fort. 



Claims 

1. Diagnostic imaging apparatus for presenting a 
three dimensional image representation, the appa- 



ratus including an image data memory (20) for stor- 
ing image data indicative of a three dimensional vol- 
ume and an image data memory accessor (76) for 
selectively accessing the stored image data, the ap- 
5 paratus comprising: a view Tenderer (182) for ren- 
dering a plurality of views (160, 162) which in total 
cover the entire visual space about a viewpoint 
(140) within the three dimensional volume; a view 
compositor (240) for combining the plurality of 
10 views into a full image (242) covering the entire vis- 
ual space about the viewpoint; a mapper (378) for 
mapping the full image into a spherical panoramic 
image; and a video processor (366) for displaying 
a portion of the spherical, panoramic image on a 
is display (370). 

2. Apparatus as claimed in claim 1 , further including a 
comparing processor (186) for comparing each pix- 
el value of one of the plurality of views with a prese- 

20 lected criteria, the comparing processor being op- 
eratively connected with the image data memory 
accessor such that the memory accessor continues 
to access pixel values of the stored memory along 
a viewing direction until a pixel value is retrieved 
25 that meets the criteria. 

3. Apparatus as claimed in claim 1 or claim 2, further 
including: a pitch control (362) to control the pitch 
of the displayed spherical panoramic image; a yaw 

30 control (362) to control the yaw of the displayed 
spherical panoramic image; and an environmental 
mapping processor (382) for controlling the video 
processor to display other portions of the spherical, 
panoramic image in accordance with the pitch and 
35 yaw controls. 

4. Apparatus as claimed in any one of claims 1 to 3, 
further including: a path motion processor (394) 
which provides sequential display of portions of a 

40 plurality of spherical, panoramic images to simulate 
movement between the plurality of images. 

5. Apparatus as claimed in any one of claims 1 to 4, 
further including a server (26) between the image 

45 data memory accessor and the view Tenderer to 
permit remote display of the panoramic image on 
the display. 

6. A method of generating an internal spherical view 
50 from a viewpoint (140) within a three dimensional 

array of image data (20) using a computer (22), the 
method comprising: generating (182) a plurality of 
two-dimensional arrays (238) which in total cover 
the entire spherical space about the viewpoint; di- 
55 viding at least one of the plurality of two-dimension- 
al arrays (254) into a plurality of first polygon arrays 
(274); scaling the plurality of first polygon arrays into 
a plurality of second polygon arrays (286); combin- 
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ing (290) the plurality of second polygon arrays and 
a portion of the plurality of two-dimensional arrays 
to form a full two-dimensional array (242) covering 
the entire spherical space about the viewpoint; 
mapping (378) the full two-dimensional array into a . s 
spherical view; and displaying at least a portion of 
the mapped, full, two-dimensional array as image 
pixels in a human-readable display (36). 

7. A method as claimed in claim 6, wherein the step io 
of generating a plurality of two-dimensional arrays 
which in total cover the entire spherical space about 

the viewpoint includes: comparing (186) each pixel 
value of at least one of the two-dimensional arrays 
with a threshold criteria; in response to a pixel value is 
failing to meet the threshold criteria, projecting 
deeper into the image data until a pixel value that 
meets the threshold criteria is identified; and insert- 
ing the pixel value that meets the threshold criteria 
into the two-dimensional array. 20 

8. A method as claimed in claim 6 or claim 7, further 
including the steps of: cropping (262) at least one 
of the plurality of the two-dimensional arrays into a 
third polygon array (264); and scaling (280) the third 25 
polygon array into a fourth polygon array (282); 
wherein the step of combining includes combining 

the fourth polygon array. 

9. A method as claimed in any one of claims 6 to 8, 30 
further including the steps of: generating (402) a 
separate two-dimensional array of an area (406) 
around the full, two-dimensional array; and display- 
ing the separate two-dimensional array on the hu- 
man-readable display. 35 

10. A method as claimed in any one of claims 6 to 9, 
further including storing the full two-dimensional ar- 
ray on a server (26) to provide remote access over 

a network (30) and remote display on the human- 40 
readable display. 
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